Device and method for machine reading comprehension question and answer

ABSTRACT

Aspects of the subject disclosure may include, systems and methods, for example, including receiving a user question data in a speech format or a text format, analyzing the user question data, selecting a plurality of documents from a plurality of domains corresponding to the user question data, searching for a plurality of passages including candidates for an answer value determined as being suitable for the user question data, in the plurality of documents, obtaining candidates by inputting the user question data and the plurality of passages into a plurality of MRC question and answer units, determining the answer value based on whether a reliability value of each of the candidates exceeds a threshold value, and providing the determined answer value to a user. Other embodiments are disclosed.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.17/989,952 filed Nov. 18, 2022, which is a continuation of U.S. patentapplication Ser. No. 16/684,902 filed Nov. 15, 2019 (now U.S. Pat. No.11,531,818). All sections of the aforementioned application(s) areincorporated herein by reference in its entirety.

FIELD OF THE DISCLOSURE

One or more embodiments relate to a machine reading comprehensionquestion and answer.

BACKGROUND

Human languages are rich, complex, and contain many vocabularies withcomplex grammar and contextual meanings, but hardware or softwareapplications generally require data to be entered in accordance withcertain formats or rules. Here, natural language input is able to beused in almost any software application for interacting with a person.Recently, in a question and answer providing method using naturallanguage, a natural language processor (NLP) module receives a lexicalinput in the form of a text or a speech and processes the lexical inputinto a form that can be processed on a computer, a processed lexicalinput passes through a context analyzer for analyzing the context of theprocessed natural language, answers are classified and finalized by adecision maker that determines an answer according to the context, and alexical output is output through a response generator for providing ananswer to a user according to the finalized answer.

SUMMARY

According to one or more embodiments, a machine reading comprehension(MRC) question and answer providing method includes receiving a userquestion data in a speech format or a text format, analyzing the userquestion data, selecting a plurality of documents from a plurality ofdomains corresponding to the user question data, searching for aplurality of passages including candidates for an answer valuedetermined as being suitable for the user question data, in theplurality of documents, obtaining candidates by inputting the userquestion data and the plurality of passages into a plurality of MRCquestion and answer units, determining the answer value based on whethera reliability value of each of the candidates exceeds a threshold value;and providing the determined answer value to a user.

In at least one variant, the analyzing further includes analyzing anentity name by morphological analysis of the user question data.

In another variant, the selecting the plurality of documents furtherincludes selecting a similar question word candidate group similar tothe user question data, selecting a similar question word having a highsimilarity from the similar question word candidate group by applying aword embedding, and in case that there is a first answer valuecorresponding to the similar question word, providing the first answervalue to a user.

In further another variant, the method further includes obtaining N*Mcandidates by inputting the user question data and the plurality ofpassages into the plurality of MRC question and answer units, where N isa number of the plurality of passages, and M is a number of theplurality of MRC question and answer units.

In further another variant, the determining further comprisesdetermining a plurality of candidates having the same reliability valuefrom among the N*M candidates as the answer value.

According to one or more embodiments, a machine reading comprehension(MRC) question and answer providing device includes a processing systemincluding a processor and a memory that stores executable instructionsthat, when executed by the processing system, facilitate performance ofoperations. The operations include receiving a user question data in aspeech format or a text format, analyzing the user question data,selecting a plurality of documents from a plurality of domainscorresponding to the user question data, searching for a plurality ofpassages including candidates for an answer value determined as beingsuitable for the user question data, in the plurality of documents,obtaining candidates by inputting the user question data and theplurality of passages into a plurality of MRC question and answer units,determining the answer value based on whether a reliability value ofeach of the candidates exceeds a threshold value; and providing thedetermined answer value to a user.

According to one or more embodiments, a machine reading comprehension(MRC) question and answer providing method includes receiving a userquestion data in a speech format or a text format, recognizing an entityname by morphological analysis of user question data, analyzing anintention of the user question data by analyzing a lexical meaning basedon the entity name, extracting a query by restoring an abbreviation wordor a substitute word of the intention of the user question data, mappinga plurality of domains corresponding to the query based on a rule-baseddomain classifier, selecting a plurality of documents corresponding tothe query based on the plurality of domains; and searching for aplurality of passages, sentence by sentence in the plurality ofdocuments and obtaining candidates by applying a plurality of MRCquestion and answer algorithms to the plurality of passages.

According to one or more embodiments, a machine reading comprehension(MRC) question and answer providing device includes a receiverconfigured to receive a user question data in a speech format or a textformat, an analyzer configured to recognize an entity name bymorphological analysis of user question data, analyze an intention ofthe user question data by analyzing a lexical meaning based on theentity name, extract a query by restoring an abbreviation word or asubstitute word of the intention of the user question data, map aplurality of domains corresponding to the query based on a rule-baseddomain classifier, a passage searcher configured to select a pluralityof documents corresponding to the query based on the plurality ofdomains, search for a plurality of passages, sentence by sentence in theplurality of documents, obtain candidates by applying a plurality of MRCquestion and answer algorithms to the plurality of passages.

One or more embodiments are to provide a best answer that is mostsuitable for a user question input in natural language.

One or more embodiments include a method of using at least one passage,document, and a machine reading and comprehension (MRC) question andanswer algorithm.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments of the disclosure.

According to one or more embodiments, a machine reading comprehension(MRC) question and answer providing method includes receiving a userquestion in the form of a speech or a text, analyzing an intention ofthe user question by analyzing lexical meanings based on an entity namerecognized through a morphological analysis of the user question,extracting a query by restoring an abbreviation or a substitute word,and mapping at least one domain information corresponding to the querybased on a rule-based domain classifier, selecting a similar questionword candidate group, selecting one similar question word having a highsimilarity from the similar question word candidate group by applyingword embedding, and, when there is a best answer value corresponding tothe selected similar question word, providing the existing best answervalue as an answer to the user; when there is no best answer valuecorresponding to the similar question word, selecting at least onedocument corresponding to the question based on the at least one mappeddomain information; searching for a plurality of passages sentence bysentence in the at least one document and applying at least one MRCquestion and answer algorithm to the plurality of retrieved passages;and, after applying the at least one MRC question and answer algorithmto the plurality of passages, selecting one best answer value based onreliability of the at least one MRC question and answer algorithm orconsistency of result values and providing the best answer value as ananswer to a user, or, when there is no best answer value, providinginformation indicating no result as an answer to the user.

According to one or more embodiments, a machine reading comprehension(MRC) question and answer providing method includes receiving a userquestion; analyzing the user question; selecting at least one documentfrom at least one domain corresponding to an analyzed user question andsearches for a passage, which is a candidate answer determined as beingsuitable for the user question, in the selected at least one document;obtaining at least one correct answer candidate value by inputting theuser question and a corresponding passage into each of at least one MRCquestion and answer unit; determining whether the at least one correctanswer candidate value is a best answer; and providing a best answervalue, which is a correct answer candidate value determined as a correctanswer, or information indicating no result as an answer to a user.

According to one or more embodiments, a machine reading comprehension(MRC) question and answer providing method includes receiving a userquestion in the form of a speech or a text; analyzing a user questionbased on a natural language processing analysis and a natural languagecomprehension analysis; during the analysis of the user question, when asimilar question word candidate group is selected, a termfrequency-inverse document frequency (TF-IDF) value is calculated withrespect to the similar question word candidate group, one similarquestion word corresponding to a high TF-IDF value is selected, andthere is a best answer value corresponding to the selected similarquestion word, providing the existing best answer value as an answer tothe user; when there is no best answer value corresponding to thesimilar question word, selecting a plurality of documents in at leastone domain corresponding to the user question and extracting a pluralityof passages that are answer candidates determined as being suitable forthe user question from the selected plurality of documents; applying aplurality of MRC question and answer algorithms are applied to theplurality of passages to obtain a plurality of correct answer candidatevalues; and determining whether there is a best answer value that isdetermined to be a correct answer from among the plurality of correctanswer candidate values and, when there is a best answer value,providing the best answer value as an answer, and when there is no bestanswer value, providing information indicating no result as an answer toa user.

According to one or more embodiments, a machine reading comprehension(MRC) question and answer providing device includes a receiverconfigured to receive a user question; an analyzer configured to analyzethe user question; a passage searcher configured to select at least onedocument from at least one domain corresponding to an analyzed userquestion and searches for a passage, which is a candidate answerdetermined as being suitable for the user question, in the selected atleast one document; a MRC question and answer unit configured to outputat least one correct answer candidate value by receiving inputs of theuser question and a corresponding passage and applying at least one MRCquestion and answer algorithm; a screener configured to determinewhether the at least one correct answer candidate value is a bestanswer; and an answer unit configured to provide a best answer value,which is a correct answer candidate value determined as a correctanswer, or information indicating no result as an answer to a user.

According to one or more embodiments, a machine reading comprehension(MRC) question and answer providing device includes a receiverconfigured to receive a user question in the form of a speech or a text;an analyzer configured to analyze a user question based on a naturallanguage processing analysis and a natural language comprehensionanalysis; a similar question matching unit configured to, during theanalysis of the user question, select a similar question word candidategroup, calculate a term frequency-inverse document frequency (TF-IDF)value with respect to the similar question word candidate group, selectone similar question word corresponding to a high TF-IDF value, and,when there is a best answer value corresponding to the selected similarquestion word, provide the existing best answer value as an answer tothe user; a passage searcher configured to, when there is no best answervalue corresponding to the similar question word, select a plurality ofdocuments in at least one domain corresponding to the user question andextract a plurality of passages that are answer candidates determined asbeing suitable for the user question from the selected plurality ofdocuments; a MRC question and answer unit configured to apply aplurality of MRC question and answer algorithms to the plurality ofpassages to obtain a plurality of correct answer candidate values; and ascreener configured to determine whether there is a best answer valuethat is determined to be a correct answer from among the plurality ofcorrect answer candidate values; and, an answer unit configured to, whenthere is a best answer value, provide the best answer value as ananswer, and when there is no best answer value, provide informationindicating no result as an answer to a user.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a system diagram showing the operation of a device on which amethod of providing machine reading comprehension (MRC) question andanswer is implemented according to an embodiment;

FIG. 2 is a schematic diagram of MRC question and answer providingdevices according to embodiments;

FIG. 3 is a schematic diagram of MRC question and answer providingdevices according to another embodiments;

FIG. 4 is a schematic diagram of an MRC question and answer algorithmaccording to an embodiment;

FIG. 5 is a diagram showing an example that a passage searcher searchesfor a passage and provides a retrieved passage to an MRC question andanswer unit;

FIG. 6 is a flowchart showing a process for determining a best answer ina screener according to an embodiment;

FIG. 7 is a diagram showing an example of performing a semantictriple-based search according to an embodiment;

FIG. 8 is a flowchart of method of providing MRC question and answeraccording to embodiments;

FIG. 9 is a flowchart of method of providing MRC question and answeraccording to another embodiment; and

FIG. 10 is a flowchart of method of providing MRC question and answeraccording to further another embodiment.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings, wherein like referencenumerals refer to like elements throughout. In this regard, the presentembodiments may have different forms and should not be construed asbeing limited to the descriptions set forth herein. Accordingly, theembodiments are merely described below, by referring to the figures, toexplain aspects of the present description. As used herein, the term“and/or” includes any and all combinations of one or more of theassociated listed items. Expressions such as “at least one of,” whenpreceding a list of elements, modify the entire list of elements and donot modify the individual elements of the list.

FIG. 1 is a system diagram showing the operation of a device on which amethod of providing machine reading comprehension (MRC) question andanswer is implemented according to an embodiment.

According to an embodiment, a device of which a method of providing MRCquestion and answer is implemented includes a terminal, a smart phone, asmart TV, an artificial intelligence (Al) speaker 114, a laptopcomputer, a tablet computer, a wearable device 118, a handheld device110, and an Al robot 112 and further includes a server 120. The deviceon which the method of providing MRC question and answer is implementedalso includes both a memory having stored therein a machine learningalgorithm library and a natural language processing and comprehendingalgorithm library and a processor for processing correspondinglibraries.

FIGS. 2 and 3 are schematic diagrams of MRC question and answerproviding devices according to embodiments. Like reference numerals inthe drawings denote the same or similar elements throughout the severalaspects.

FIG. 2 is a schematic diagram of an MRC question and answer providingdevice 200 according to an embodiment.

The MRC question and answer providing device 200 may include a receiver210, an analyzer 220, a passage searcher 230, an MRC question and answerunit 240, a screener 250, and an answer unit 260 and may further includean administrator setting unit (not shown).

The receiver 210 receives a user question, the analyzer 220 analyzes theuser question, the passage searcher 230 selects at least one documentfrom at least one domain corresponding to the analyzed user question,and a passage, which is a candidate answer determined as being suitablefor the user question, is searched for in the at least one selecteddocument. In this case, N passages (N is a natural number) may beretrieved. An example of searching for a passage is shown in FIG. 5 .

Each of at least one MRC question and answer unit 240 receives a userquestion and a set of N passages corresponding to the user question asinputs and obtains N*M correct answer candidate values by applying M (Mis a natural number) MRC algorithms to each of the N passages. Anexample of characteristics of an MRC algorithm is shown in FIG. 4 .

The screener 250 determines a best answer value from among the N*Mcorrect answer candidate values. The answer unit 260 transmits the bestanswer value determined by the screener 250 or information indicatingthat there is no result corresponding to the user question to a user inthe form of answer information.

FIG. 3 is a schematic diagram of an MRC question and answer providingdevice 300 according to another embodiment.

The MRC question and answer providing device 300 may include a receiver310, an analyzer 320, a passage searcher 330, an MRC question and answerunit 340, a screener 350, and an answer unit 360 and may further includean administrator setting unit (not shown). The analyzer 320 furtherincludes a similar question matching unit 322. The screener 350 furtherincludes a consistency checker 352 and a reliability checker 354.

The administrator setting unit stores a best answer value determined bythe screener 350 as a semantic triple and manages settings of thesimilar question matching unit 332, the passage searcher 330, and theMRC question and answer unit 340. Also, the administrator setting unitmay store all information generated by the receiver 310, the analyzer320, the passage searcher 330, the MRC question and answer unit 340, thescreener 350, and the answer unit 360. Examples of all the informationinclude system logs, a plurality of passages, a plurality of correctanswer candidate values, the reliability of each of the correct answercandidate values, a used MRC question and answer algorithm, a passageretrieval result, a passage retrieval time, and whether a correct answeris determined by the screener 350.

The receiver 310 may receive a user question in the form of a speech ora text. The reception of a user question includes both a case where aperson asks a question by using a natural language-based sentencethrough a speech recognition speaker or a case where a person asks aquestion by typing through a chatbot. When a speech is recognizedthrough a speech recognition speaker, the receiver 310 converts thevoice into a text through a speech-to-text (STT) conversion. Forexample, when a user is a fan of the singer ‘Shinee’, the receiver 310may receive a question “what is the name of the album that Shineereleased this time?”

The analyzer 320 receives a user question in the form of a text throughthe receiver 310 and then analyzes the user question through naturallanguage processing and natural language comprehension. The analyzer 320may analyze the user question based on a morphological analysis,recognize the entity name, analyze a lexical meaning based on therecognized entity name, analyze the intention of the user question,extract a query by restoring an abbreviation or a substitute word, andmap at least one domain information corresponding to the query based ona rule-based domain classifier. According to an embodiment, at least onedomain information corresponding to a query refers to a pre-writtendomain-specific document.

During a natural language comprehension process, the analyzer 320analyzes which domain the user question belongs to and what the corecontent of the user question is and transmits a result of the analysisto the passage searcher 330. For example, for the question “what is thename of the album that Shinee released this time?” received by thereceiver 310, the analyzer 320 maps the question to domains “music” and“person”. In this case, the user question may be considered as not beingdirectly related to domains “movie” and “economy”. In this case, thepassage searcher 330, which received a result of an analysis, may searchfor the user question in domains related to “music” and “person” andexcludes domains like “movie” and “economy”, thereby improving thesearching speed and the accuracy of documents. Also, the analyzer 320recognizes “Shinee”, “this time”, and “album name” as entity names orkey words based on a morphological analysis.

When the analyzer 320 analyzes a user question by using deep learningtechnology, the similar question matching unit 322 selects a similarquestion word candidate group, selects one similar question word withhigh similarity by applying word- embedding to the similar question wordcandidate group, and, when a best answer value corresponding to thesimilar question word already exists, provides the existing best answervalue to a user as an answer.

For example, the similar question matching unit 322 may applyword-embedding to the question “What is the name of the album thatShinee released latest?” and select a similar question “Tell me the nameof Shinee's latest album” having a similar meaning. When a best answervalue “‘The Story of Light’ EP3” for the similar question “Tell me thename of Shinee's latest album” already exists, the similar questionmatching unit 322 may replace the user question “What is the name of thealbum that Shinee released latest?” to “Tell me the name of Shinee'slatest album”, which is a representative question with high similarity,and provide the existing best answer value as an answer to the userquestion.

The passage searcher 330 selects at least one document from at least onedomain corresponding to an analyzed user question and searches for apassage, which is a candidate answer determined as being suitable forthe user question, in the selected at least one document. In this case,the at least one document includes news articles, dictionaries, and webdocuments such as blogs. FIG. 5 is a diagram showing an example that apassage searcher searches for a passage and provides a retrieved passageto an MRC question and answer unit. Referring to FIG. 5 , an example inwhich the passage searcher 520 extracts five passages 540 a, 540 b, 540c, 540 d, and 540 e in relation to a user question “What is the name ofthe album that Shinee released latest?” and transmits the passages 540a, 540 b, 540 c, 540 d, and 540 e to the MRC question and answer unit340 is shown.

The passage searcher 330 may use key words “Shinee”, “this time”, and“album name”, which are recognized by the analyzer 320 in domainsrelated to “music” and “person”, to search for passages.

The passage searcher 330 utilizes a term frequency-inverse documentfrequency (TF-IDF) and searches for N passages in at least one documentin the descending order of TF-IDF values. In this case, N passages maybe extracted from one document or N passages may be extracted from aplurality of documents. According to an embodiment, the passage searcher330 may extract at least one passage in the descending order of TF-IDFvalues having values equal to or greater than a predetermined score.

The TF-IDF is a weight used for information retrieval and text miningand is a statistical value that indicates how important a word is withina particular document when there are groups of documents. The TF-IDF maybe used to extract key words of a document, to determine rankings ofsearch results in a search engine, or to determine similarity betweendocuments.

A term frequency (TF) is a value that indicates how often a particularword appears within a document. The higher the TF is, the more importantthe document may be considered as being. However, when a word isfrequently used in a group of documents, it means that the word iscommon. A frequency that the word is used is referred to as a documentfrequency (OF), and the inverse thereof is called inverse documentfrequency (IDF). A TF-IDF is the product of a TF and an IDF.

An IDF value depends on the background of a document group. For example,the word “atoms” may have a high IDF value and become a key word indocuments because the word does not frequently appear in ordinarydocuments. However, the word becomes a frequently used word in a groupof documents related to atoms, and thus other words that may be used toclassify and distinguish the documents receive high weights.

According to an embodiment, the passage searcher 330 calculates a TF-IDFvalue based on a user question, a query analyzed from the user question,and a result of a morphological analysis of the query. In this case, thenumber of passages extracted by the passage searcher 330 may beinitially set to an arbitrary number and may be automatically adjustedlater by using log-based statistical values. The log-based statisticalvalues include a result value selected as the best answer by thescreener 350 and a log analysis value recorded in the administratorsetting unit.

For example, when the administrator setting unit initially sets thepassage searcher 330 to extract five passages and an actual result isnot significantly different from that of a case of extracting threepassages, the passage searcher 330 is automatically adjusted to extractthree passages, and thus search speed and efficiency may be improved.

According to an embodiment, the passage searcher 330 may also set anextraction range for extracting passages. The administrator setting unitmay initially set the passage searcher 330 to extract passages in unitsof sentences exceeding a predetermined byte. For example, theadministrator setting unit may initially set the passage searcher 330 toextract a passage within the range from 500 bytes to 800 bytes includingKorean characters and spaces. A passage is extracted sentence bysentence according to periods. For example, to extract a passagesentence by sentence, the passage is extracted sentence by sentenceincluding periods, even when the passage exceeds 800 bytes.

The passage searcher 330 may automatically adjust an extraction rangefor extracting a passage based on the statistics that correct answercandidate values determined by the MRC question and answer algorithmused by each of at least one MRC question and answer unit 340 aredetermined as best answers. For example, when the initial setting wasmade to extract passages within the range from 500 bytes to 800 bytesand the statistics of the byte values of passages determined as bestanswers in the screener 350 is from 600 to 630 bytes, the administratorsetting unit may feedback this value to the passage searcher 330 andadjust the extraction range for extracting passages to from 600 to 630bytes.

According to an embodiment, the MRC question and answer device 300includes one or a plurality of MRC question and answer units 340,wherein each MRC question and answer unit 340 is implemented to apply anMRC question and answer algorithm to passages.

FIG. 4 is a diagram showing an implementation example of an MRC questionand answer algorithm 400. The MRC question and answer algorithm refersto an algorithm that a machine comprehends a user question includingnatural language and automatically reads and derives an answer frompassages including an answer corresponding to the user question. The MRCquestion and answer algorithm is implemented based on a recurrent neuralnetwork (RNN) based attention model 410 to repeatedly read documents tosearch for a correct answer and select a proposal, a user question, anda correct answer as a group of knowledge base candidates. Examples ofthe MRC question and answer algorithms include neural networkalgorithms, such as an R-Net, a BiDAF, and a ReasoNet model.

According to an embodiment, the MRC question and answer unit 340generates N*M correct answer candidate values by applying M MRC questionand answer algorithms to N passages corresponding to one user questionand transmits the N*M correct answer candidate values and reliabilityinformation indicating the probability of being a correct answer foreach of the N*M correct answer candidate values to the screener 350. Forexample, when three passages are extracted for one user question andthree MRC question answering algorithms are applied, a total of ninecorrect answer candidate values are derived. The MRC question and answerunit 340 transmits the nine correct answer candidate values andreliability information indicating the probability of being a correctanswer for each of the nine correct answer candidate values to thescreener 350.

The screener 350 determines one best answer according to a criterionpre-set from among the correct answer candidate values for the userquestion. The method of determining the best answer in the screener 350is described in detail with reference to FIG. 6 .

When there is a plurality of correct answer candidate values, thescreener 350 first checks whether the majority of or at least a certainpercentage of the plurality of correct answer candidate values areconsistent through the consistency checker 352 (operation S630). To thisend, the consistency checker 352 may perform pre-processing operationsfor analyzing at least one correct answer candidate value for periods,spacing, and case sensitivity on the basis of morphological analysis,unifying the formats of the at least one correct answer candidate valuesaccording to pre-set criteria, and also restoring abbreviations orsubstitute terms. When the majority of or at least a certain percentageof the plurality of correct answer candidate values are consistent, theconsistency checker selects the corresponding correct answer candidatevalues as best answers (operation S632).

For example, when there are nine correct answer candidate values for thequestion “What is the name of the album that Shinee released latest?”and five or more correct answer candidate values including ‘the story oflight ep3, ‘The Story of Light EP3’, THE STORY OF LIGHT', etc.consistently indicate ‘The Story of Light EP3’, the correct answercandidate value ‘The Story of Light EP3’ is selected as the best answer.

When a best answer is not selected by the consistency checker 352, thescreener 350 determines whether there is a correct answer candidatevalue corresponding to a reliability exceeding a pre-set critical value(e.g., 90%) through the reliability checker 354 (operation S640). Whenthere is a correct answer candidate value corresponding to a reliability(e.g., 95%) exceeding the preset critical value, the correspondingcorrect answer candidate value is determined as the best answer. In thiscase, when the reliability of a plurality of correct answer candidatevalues exceeds the preset critical value, the correct answer candidatevalue corresponding to the highest reliability value is determined asthe best answer (operation S642).

When the screener 350 is unable to find a best answer value through theconsistency checker 352 and the reliability checker 354, the screener350 determines whether the critical value of the reliability of thecorrect answer candidate values is less than a preset reference value(operation S650) and, when the reliability of all of the correct answercandidate values is less than the critical value, transmits ‘no result’to the answer unit 360 (operation S652).

When the screener 350 is unable to find a best answer value through theconsistency checker 352 and the reliability checker 354 and thereliability of correct answer candidate values is equal to or greaterthan a first critical value and less than a second critical value, thescreener 350 classifies the correct answer candidate values according tosimilarity of meanings based on a word-embedding value scheme,determines whether the majority of or at least a pre-set percentage ofthe correct answer candidate values classified by meanings have the samevalue, and, when the majority of or at least a pre-set percentage of thecorrect answer candidate values classified by meanings have the samevalue, selects the corresponding value as a best answer (operationS660). When the majority of or at least a pre-set percentage of thecorrect answer candidate values classified by meanings do not have thesame value, the screener 350 transmits ‘no result’ to the answer unit360.

For example, ‘The Story of Light EP3’, ‘The Story of Light EP3’, and‘The Story of Light Third EP’ are semantically the same, but a resultmay vary depending on which passage is extracted. For such a case, thesimilarity between result values is measured through similar questionword matching trained in advance and, when the similarity is equal to orhigher than a particular similarity, correct answer candidate values aredetermined as the same values.

The answer unit 360 provides a best answer or information indicating noresult transmitted from the screener 350 to a user as an answer.

FIG. 6 is a flowchart showing a process for determining a best answer ina screener according to an embodiment.

According to an embodiment, the screener receives N*M correct answercandidate values extracted by applying M MRC algorithms to N passages(operations S610 and S620) and determines whether the majority of thereceived correct answer candidate values are the same (operation S630).When the majority of the correct answer candidate values aresubstantially the same, the corresponding correct answer candidate valueis selected as a best answer value (operation S632). When the majorityof the received correct answer candidate values are not the same, it isdetermined whether there is a correct answer candidate valuecorresponding to reliability exceeds a preset critical value (operationS640). When there is a correct answer candidate value corresponding toreliability exceeds the preset critical value, the corresponding correctanswer candidate value is selected as a best answer value (operationS642).

When there is no correct answer candidate value corresponding to areliability exceeding the preset critical value, it is determinedwhether all of the correct answer candidate values are less than anotherpreset critical value and, when all of the correct answer candidatevalues are less than the other preset critical value, transmits ‘noresult’ as an answer to a user (operation S652).

When the reliability of correct answer candidate values is between thefirst threshold value and the second threshold value, the correct answercandidate values are classified according to similarity of meaningsbased on a word-embedding value scheme and, when the majority of thecorrect answer candidate values classified by meanings have the samevalue, the corresponding value is selected as a best answer (operationS660).

FIG. 7 is a diagram showing an example of performing a semantictriple-based search according to an embodiment.

According to an embodiment, the administrator setting unit stores a bestanswer value determined by a screener based on the semantic triple. Asemantic triple database is a particular type of knowledge-baseddatabase that simulates queries of actual users, and a best answer maybe searched for without any additional reasoning. The semantic tripledatabase has the form of entity 732-attribute 734-instant answer 738.

In FIG. 7 , when a user question 710 “What is the height of Mt. Baekdu?”Is received, key words “Mt. Baekdu” and “height” may be extracted and“Mt. Baekdu” may be analyzed as an entity of the user question 710 and“height” may be analyzed as an intention of the user question 710.

The administrator setting unit checks a semantic triple database (DB)730, searches for data with entity=“baekdusan” and attribute=“height”,determines an instant answer of the corresponding items as a result, andprovides a corresponding answer 750 of 2,744 m to a user.

The administrator setting unit checks the forms of the user question andthe best answer to store a best answer value determined by the screenerbased on the semantic triple. In this case, a question transformationprocess includes a natural language comprehension technique and asemantic triple entity/attribute data retrieval technique.

FIG. 8 is a flowchart of a method of providing MRC question and answeraccording to an embodiment.

According to an embodiment, in a method of providing MRC question andanswer, a user question is received and analyzed (operation S810 andoperation S820), at least one document is selected in at least onedomain based on the analyzed question, and a passage is searched for inthe selected document (operation S830).

At least one correct answer candidate value is obtained by applying atleast one MRC question and answer algorithm to each retrieved passage(operation S840), it is determined whether the obtained at least onecorrect answer candidate value is a correct answer (operation S850), anda best answer value or no result is provided as an answer to a user(operation S860).

FIG. 9 is a flowchart of a method of providing MRC question and answeraccording to another embodiment.

According to an embodiment, in a method of providing MRC question andanswer, a user question is received in the form of a speech, or a textand the user question is analyzed based on a natural language processinganalysis and a natural language comprehension analysis (operations S910and S920).

When one similar question word is selected from a similar question wordcandidate group selected through a similar question word matching schemeduring the analysis of a user question and a best answer valuecorresponding to the selected similar question word already exists, theexisting best answer value is provided as an answer to a user (operationS930).

When there is no best answer value corresponding to the similar questionword, a plurality of documents is selected in at least one domaincorresponding to the user question and a plurality of passages areextracted from the selected plurality of documents (operation S940).Thereafter, a plurality of MRC question and answer algorithms areapplied to the plurality of passages to obtain a plurality of correctanswer candidate values (operation S950). It is determined whether thereis a best answer value from among the plurality of correct answercandidate values. When there is a best answer value, the best answervalue is provided as an answer to a user. When there is no best answervalue, information indicating no result is provided to the user(operations S960 and S970).

FIG. 10 is a flowchart of a method of providing MRC question and answeraccording to another embodiment.

According to an embodiment, in a method of providing MRC question andanswer, a user question is received in the form of a speech or a text(operation S1010). Thereafter, the intention of the user question isanalyzed by analyzing lexical meanings based on the entity namerecognized through a morphological analysis of the user question, aquery is extracted by restoring an abbreviation or a substitute word,and at least one domain information corresponding to the query is mappedbased on a rule-based domain classifier (operation S1020).

During the analysis of the user question, when a similar question wordcandidate group is selected, one similar question word having a highsimilarity is selected from the similar question word candidate group byapplying word embedding, and, when there is a best answer valuecorresponding to the selected similar question word, the existing bestanswer value is provided as an answer to the user (operation S1030).

When there is no best answer value corresponding to the similar questionword, at least one document corresponding to the question is selected ineach of the at least one domain based on the at least one mapped domaininformation (operation S1040).

Then, after searching for a plurality of passages sentence by sentencein the at least one document and applying at least one MRC question andanswer algorithm to the plurality of retrieved passages (operationS1050), one best answer value is selected based on the reliability ofderived correct answer candidates or the consistency of the correctanswer candidates and provided as an answer to the user. When there isno best answer value, information indicating no result is provided as ananswer to the user (operation S1060).

The apparatus described above may be implemented as a hardwarecomponent, a software component, and/or a combination of hardwarecomponents and software components. For example, the devices andcomponents described in the embodiments may be implemented by using oneor more general purpose or special purpose computers, such as aprocessor, a controller, an arithmetic logic unit (ALU), a digitalsignal processor, a microcomputer, a field programmable gate array(FPGA), a programmable logic unit (PLU), a microprocessor, or any otherdevice capable of executing and responding to instructions. A processingdevice may execute an operating system (OS) and one or more softwareapplications running on the OS. The processing device may also access,store, manipulate, process, and generate data in response to executionof software. For the convenience of explanation, it has been describedabove that one processing device is used. However, it would be obviousto one of ordinary skill in the art that the processing device mayinclude a plurality of processing elements and/or a plurality of typesof processing elements. For example, the processing device may include aplurality of processors or one processor and one controller. Also, otherprocessing configurations like parallel processors may be employed.

The software may include a computer program, code, instructions, or acombination of one or more of the foregoing, to configure the processingdevice to operate as demanded or to command the processing deviceindependently or collectively. For the purpose of interpreting orproviding instructions or data to the processing device, software and/ordata may be permanently or temporarily embodied in any type of machine,component, physical device, virtual equipment, computer storage medium,or a signal wave to be transmitted. The software may be distributed overnetworked computer systems so that they may be stored or executed in adistributed manner. The software and data may be stored on one or morecomputer-readable recording media.

The methods according to embodiments may be embodied in the form ofprogram instructions that can be executed by various computer means andrecorded on a computer-readable recording medium. The computer-readablerecording media may include program instructions, data files, and datastructures alone or a combination thereof. The program commands recordedon the medium may be specially designed and configured for exampleembodiments or may be published and available to one of ordinary skillin computer software. Examples of the computer-readable recording mediuminclude a hardware device specially configured to store and performprogram instructions, for example, a magnetic medium, such as a harddisk, a floppy disk, and magnetic tape, an optical recording medium,such as a CD-ROM, a DVD, and the like, a magneto-optical medium, such asa floptical disc, ROM, RAM, a flash memory, and the like. Examples ofprogram commands include machine language code such as code generated bya compiler, as well as high-level language code that may be executed bya computer using an interpreter or the like. The hardware devicedescribed above may be configured to operate as one or more softwaremodules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodimentsand the drawings as described above, various modifications andvariations are possible to one of ordinary skill in the art from theabove description. For example, the described techniques may beperformed in a different order than the described method, and/orcomponents of the described systems, structures, devices, circuits, etc.may be combined or combined in a different manner than the describedmethod, or other components. Or even if replaced or substituted byequivalents, an appropriate result can be achieved.

According to an embodiment, a method of providing MRC question andanswer may provide a best answer by understanding the meaning of auser's question and based on the meaning, applying at least one MRCquestion and answer algorithm to at least one document and at least onepassage associated with a searched domain.

It should be understood that embodiments described herein should beconsidered in a descriptive sense only and not for purposes oflimitation. Descriptions of features or aspects within each embodimentshould typically be considered as available for other similar featuresor aspects in other embodiments. While one or more embodiments have beendescribed with reference to the figures, it will be understood by thoseof ordinary skill in the art that various changes in form and detailsmay be made therein without departing from the spirit and scope of thedisclosure as defined by the following claims.

What is claimed is:
 1. A machine reading comprehension (MRC) questionand answer providing method comprising: receiving, by a processingsystem including a processor, a user question data in a speech format ora text format; analyzing, by the processing system, the user questiondata; selecting, by the processing system, a plurality of documents froma plurality of domains corresponding to the user question data;searching, by the processing system, in the plurality of documents, fora plurality of passages including candidates for an answer valuedetermined as being suitable for the user question data; obtaining, bythe processing system, the candidates by inputting the user questiondata and the plurality of passages into a plurality of MRC question andanswer units, determining, by the processing system, the answer valuebased on whether a reliability value of each of the candidates exceeds athreshold value; and providing, by the processing system, the answervalue to a user.
 2. The MRC question and answer providing method ofclaim 1, wherein the analyzing further comprises analyzing an entityname by morphological analysis of the user question data.
 3. The MRCquestion and answer providing method of claim 1, wherein the selectingthe plurality of documents further comprises: selecting a similarquestion word candidate group similar to the user question data;selecting a similar question word having a high similarity from thesimilar question word candidate group by applying a word embedding; andin case that there is a first answer value corresponding to the similarquestion word, providing the first answer value to a user.
 4. The MRCquestion and answer providing method of claim 1, wherein, in theobtaining, obtaining N*M candidates by inputting the user question dataand the plurality of passages into the plurality of MRC question andanswer units, wherein N is a number of the plurality of passages, and Mis a number of the plurality of MRC question and answer units.
 5. TheMRC question and answer providing method of claim 4, wherein thedetermining further comprises determining a plurality of candidateshaving the same reliability value from among the N*M candidates as theanswer value.
 6. A machine reading comprehension (MRC) question andanswer device comprising: a processing system including a processor; anda memory that stores executable instructions that, when executed by theprocessing system, facilitate performance of operations, the operationscomprising: receiving a user question data in a speech format or a textformat; analyzing the user question data; selecting a plurality ofdocuments from a plurality of domains corresponding to the user questiondata; searching for a plurality of passages including candidates for ananswer value determined as being suitable for the user question data, inthe plurality of documents; obtaining candidates by inputting the userquestion data and the plurality of passages into a plurality of MRCquestion and answer units; determining the answer value based on whethera reliability value of each of the candidates exceeds a threshold value;and providing the determined answer value to a user.
 7. The MRC questionand answer device of claim 6, wherein, in the analyzing operation, anentity name is recognized by morphological analysis of the user questiondata.
 8. The MRC question and answer device of claim 6, wherein theoperations further comprise: selecting a similar question word candidategroup similar to the user question data, selecting a similar questionword having a high similarity from the similar question word candidategroup by applying a word embedding, and providing a first answer valueto a user upon determination that there is the first answer valuecorresponding to the similar question word.
 9. The MRC question andanswer device of claim 6, wherein, in the obtaining operation, theprocessing system obtains N*M candidates by inputting the user questiondata and the plurality of passages into a plurality of MRC question andanswer units, wherein N is a number of the plurality of passages, and Mis a number of the plurality of MRC question and answer units.
 10. TheMRC question and answer device of claim 9, wherein, in the determiningoperation, a plurality of candidates having the same reliability valueis determined from among the N*M candidates as the answer value.